{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In mathematics, a time series is a series of data points indexed (or listed or graphed) in time order. Most commonly, a time series is a sequence taken at successive equally spaced points in time - [Wikipedia](https://en.wikipedia.org/wiki/Time_series). For any types of machine learning time series tasks, getting the correct data format is important. Below, we will introduce what and how to generate the correct time series data format based on different tasks.  \n",
    "\n",
    "\n",
    "1. Forecasting\n",
    "2. Classification and Regression\n",
    "3. Anomaly detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Forecasting Data Format\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The required input data format is a two-dimensional structure (``pandas.DataFrame``), which should contain a TimeStamp column (``time_col``) and one or more variable columns (``var_col_0``, ``var_col_1``, ``var_col_2``,... ``var_col_n``,). See example below: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1 Data Format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```python\n",
    "     time_col          var_col_0        var_col_1        var_col_2     ...    var_col_n\n",
    "xxxx-xx-xx xx:xx:xx        x                x                x                    x\n",
    "xxxx-xx-xx xx:xx:xx        x                x                x                    -\n",
    "xxxx-xx-xx xx:xx:xx        x                x                x                    x\n",
    "xxxx-xx-xx xx:xx:xx        -                x                x                    x\n",
    "xxxx-xx-xx xx:xx:xx        x                x                x                    x\n",
    "xxxx-xx-xx xx:xx:xx        x                -                x                    x\n",
    "xxxx-xx-xx xx:xx:xx        x                -                x                    x\n",
    "xxxx-xx-xx xx:xx:xx        x                x                -                    x\n",
    "xxxx-xx-xx xx:xx:xx        x                x                x                    x\n",
    "xxxx-xx-xx xx:xx:xx        x                x                -                    x\n",
    "xxxx-xx-xx xx:xx:xx        -                -                -                    -\n",
    "xxxx-xx-xx xx:xx:xx        x                x                x                    x\n",
    "        -                  -                -                -                    -\n",
    "xxxx-xx-xx xx:xx:xx        x                x                x                    x\n",
    "```\n",
    "\n",
    "where *xxxx-xx-xx xx:xx:xx* stands for datetime. Ideally, the date format is ``YYYY-MM-DD`` and the time is ``HH:MM:SS``. In variable columns, (*x*) represents a certain value and (*-*) means the value is missing. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "  - HyperTS expects must also contain time columns, column names are not specified, whether ds, TS, TIMESTAMP, timestamp or other. The datetime can be any format as long as it can be identified by ``pandas.to_datetime``.  \n",
    "  - HyperTS could sort the data in time sequence if the time is random. \n",
    "  - HyperTS supports input data with various time frequencies. For example, second(S)、minute(T)、hour(H)、day(D)、week(W)、month(M)、year(Y),etc.\n",
    "  - HyperTS could impute the missing time points and segments during the preprocessing stage.\n",
    "  - HyperTS could dropout the repeated rows during the preprocessing stage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, there are extra variables generated during the modeling, which are called covariates. They will be added parallel to the input variables. See example below:  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "     time_col          var_col_0   var_col_1 ... var_col_n     covar_col_0    covar_col_1 ... covar_col_m\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x \n",
    "xxxx-xx-xx xx:xx:xx        x          x              -              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              -               x\n",
    "xxxx-xx-xx xx:xx:xx        -          x              x              x              x               -\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        x          -              x              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        x          -              x              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              -               x\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        -          -              -              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x\n",
    "        -                  -          -              -              -              -               -\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x\n",
    "```\n",
    "where *covar_col_i (i=1, 2, .., m)*  stand for covariates。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "  - Covariates can be continuous values or discrete values. \n",
    "  - Covariates can contain repeated and missing values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2 Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.2.1 Generate a Random Dataset without Covariates "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>val_0</th>\n",
       "      <th>val_1</th>\n",
       "      <th>val_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2022-02-01 00:00:00</td>\n",
       "      <td>0.599713</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.538076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2022-02-01 01:00:00</td>\n",
       "      <td>2.199247</td>\n",
       "      <td>0.2</td>\n",
       "      <td>-0.007808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2022-02-01 02:00:00</td>\n",
       "      <td>-0.316841</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.680783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2022-02-01 03:00:00</td>\n",
       "      <td>-0.747250</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.203446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2022-02-01 04:00:00</td>\n",
       "      <td>-0.404355</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.922243</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timestamp     val_0  val_1     val_2\n",
       "0 2022-02-01 00:00:00  0.599713    0.5  1.538076\n",
       "1 2022-02-01 01:00:00  2.199247    0.2 -0.007808\n",
       "2 2022-02-01 02:00:00 -0.316841    NaN  0.680783\n",
       "3 2022-02-01 03:00:00 -0.747250    0.9  0.203446\n",
       "4 2022-02-01 04:00:00 -0.404355    0.0 -0.922243"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "size=5\n",
    "\n",
    "# without covariables\n",
    "df_no_covariates = pd.DataFrame({\n",
    "    'timestamp': pd.date_range(start='2022-02-01', periods=5, freq='H'),\n",
    "    'val_0': np.random.normal(size=size),\n",
    "    'val_1': [0.5, 0.2, np.nan, 0.9, 0.0],\n",
    "    'val_2': np.random.normal(size=size),\n",
    "})\n",
    "\n",
    "df_no_covariates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output shows that:\n",
    "\n",
    "- The name of the timestamp column is 'timestamp';\n",
    "- The names of the target columns are 'var_0',  'var_1',  'var_2';\n",
    "- The time frequency is per hour: 'H';\n",
    "- The dataset contain missing values;\n",
    "- It's a multivariate timeseries forecasting task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.2.2 Generate a Random Dataset with Covariates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>val_0</th>\n",
       "      <th>val_1</th>\n",
       "      <th>val_2</th>\n",
       "      <th>covar_0</th>\n",
       "      <th>covar_1</th>\n",
       "      <th>covar_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2022-02-01</td>\n",
       "      <td>-0.921455</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>a</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2022-02-02</td>\n",
       "      <td>-0.300891</td>\n",
       "      <td>52.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.4</td>\n",
       "      <td>a</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2022-02-03</td>\n",
       "      <td>0.245227</td>\n",
       "      <td>34.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.2</td>\n",
       "      <td>b</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2022-02-04</td>\n",
       "      <td>1.299430</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.7</td>\n",
       "      <td>b</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2022-02-05</td>\n",
       "      <td>-0.131246</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>b</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   timestamp     val_0  val_1  val_2  covar_0 covar_1  covar_2\n",
       "0 2022-02-01 -0.921455   12.0    0.5      0.2       a      1.0\n",
       "1 2022-02-02 -0.300891   52.0    0.2      0.4       a      2.0\n",
       "2 2022-02-03  0.245227   34.0    NaN      0.2       b      2.0\n",
       "3 2022-02-04  1.299430    NaN    0.9      0.7       b      NaN\n",
       "4 2022-02-05 -0.131246  100.0    0.0      0.1       b      3.0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# with covariables\n",
    "df_with_covariates = pd.DataFrame({\n",
    "    'timestamp': pd.date_range(start='2022-02-01', periods=size, freq='D'),\n",
    "    'val_0': np.random.normal(size=size),\n",
    "    'val_1': [12, 52, 34, np.nan, 100],\n",
    "    'val_2': [0.5, 0.2, np.nan, 0.9, 0.0],\n",
    "    'covar_0': [0.2, 0.4, 0.2, 0.7, 0.1],\n",
    "    'covar_1': ['a', 'a', 'b', 'b', 'b'],\n",
    "    'covar_2': [1, 2, 2, None, 3], \n",
    "})\n",
    "\n",
    "df_with_covariates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output shows that:\n",
    "\n",
    "- The name of the timestamp column is 'timestamp';\n",
    "- The names of the target columns are 'var_0',  'var_1',  'var_2';\n",
    "- The names of the covariates columns are 'covar_0',  'covar_1',  'covar_2';\n",
    "- The time frequency is per day: 'D';\n",
    "- The dataset contain missing values;\n",
    "- It's a multivariate timeseries forecasting task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.2.3 Import build-in Forecasting Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hyperts.datasets import (load_random_univariate_forecast_dataset, \n",
    "                              load_random_multivariate_forecast_dataset,\n",
    "                              load_network_traffic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ds</th>\n",
       "      <th>id</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2013-01-01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.619163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2013-01-02</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.600649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2013-01-03</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.236368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2013-01-04</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.534748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2013-01-05</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.940991</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          ds   id     value\n",
       "0 2013-01-01  1.0  0.619163\n",
       "1 2013-01-02  0.0  0.600649\n",
       "2 2013-01-03  1.0  0.236368\n",
       "3 2013-01-04  1.0  0.534748\n",
       "4 2013-01-05  0.0  0.940991"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df0 = load_random_univariate_forecast_dataset(return_X_y=False)\n",
    "df0.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this dataset, ```ds``` is the name of time column, time frequency is day (D), target variable (i.e., forecast variable) is ```value``` column, ```id``` column is covariable column, so it is a univariate time series task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ds</th>\n",
       "      <th>Var_1</th>\n",
       "      <th>Var_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2022-10-27 12:07:27.572441</td>\n",
       "      <td>0.266213</td>\n",
       "      <td>1.098948</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2022-10-28 12:07:27.572441</td>\n",
       "      <td>1.632756</td>\n",
       "      <td>1.829807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2022-10-29 12:07:27.572441</td>\n",
       "      <td>2.871863</td>\n",
       "      <td>3.123874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2022-10-30 12:07:27.572441</td>\n",
       "      <td>3.669629</td>\n",
       "      <td>3.957610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2022-10-31 12:07:27.572441</td>\n",
       "      <td>4.353291</td>\n",
       "      <td>5.086246</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          ds     Var_1     Var_2\n",
       "0 2022-10-27 12:07:27.572441  0.266213  1.098948\n",
       "1 2022-10-28 12:07:27.572441  1.632756  1.829807\n",
       "2 2022-10-29 12:07:27.572441  2.871863  3.123874\n",
       "3 2022-10-30 12:07:27.572441  3.669629  3.957610\n",
       "4 2022-10-31 12:07:27.572441  4.353291  5.086246"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = load_random_multivariate_forecast_dataset(return_X_y=False)\n",
    "df1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this dataset, ```ds``` is the name of time column, and time frequency is day(D). The target variable(i.e., forecast variable) contains two columns ```Var_1``` and ```Var_2```and there is no covariable. Therfore, it is a multivariate time series task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TimeStamp</th>\n",
       "      <th>Var_1</th>\n",
       "      <th>Var_2</th>\n",
       "      <th>Var_3</th>\n",
       "      <th>Var_4</th>\n",
       "      <th>Var_5</th>\n",
       "      <th>Var_6</th>\n",
       "      <th>HourSin</th>\n",
       "      <th>WeekCos</th>\n",
       "      <th>CBWD</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-03-01 00:00:00</td>\n",
       "      <td>0.7534</td>\n",
       "      <td>3.375</td>\n",
       "      <td>10.195</td>\n",
       "      <td>1.4490</td>\n",
       "      <td>19174.977</td>\n",
       "      <td>286443.880</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-03-01 01:00:00</td>\n",
       "      <td>0.3376</td>\n",
       "      <td>2.414</td>\n",
       "      <td>3.920</td>\n",
       "      <td>0.4065</td>\n",
       "      <td>7529.263</td>\n",
       "      <td>178930.450</td>\n",
       "      <td>0.258819</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-03-01 02:00:00</td>\n",
       "      <td>0.2032</td>\n",
       "      <td>1.654</td>\n",
       "      <td>3.318</td>\n",
       "      <td>0.2142</td>\n",
       "      <td>3310.539</td>\n",
       "      <td>42296.164</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-03-01 03:00:00</td>\n",
       "      <td>0.2420</td>\n",
       "      <td>1.393</td>\n",
       "      <td>3.148</td>\n",
       "      <td>0.2312</td>\n",
       "      <td>4535.464</td>\n",
       "      <td>26220.232</td>\n",
       "      <td>0.707107</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-03-01 04:00:00</td>\n",
       "      <td>0.1940</td>\n",
       "      <td>1.429</td>\n",
       "      <td>3.215</td>\n",
       "      <td>0.2157</td>\n",
       "      <td>2732.911</td>\n",
       "      <td>27990.348</td>\n",
       "      <td>0.866025</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NW</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             TimeStamp   Var_1  Var_2   Var_3   Var_4      Var_5       Var_6  \\\n",
       "0  2021-03-01 00:00:00  0.7534  3.375  10.195  1.4490  19174.977  286443.880   \n",
       "1  2021-03-01 01:00:00  0.3376  2.414   3.920  0.4065   7529.263  178930.450   \n",
       "2  2021-03-01 02:00:00  0.2032  1.654   3.318  0.2142   3310.539   42296.164   \n",
       "3  2021-03-01 03:00:00  0.2420  1.393   3.148  0.2312   4535.464   26220.232   \n",
       "4  2021-03-01 04:00:00  0.1940  1.429   3.215  0.2157   2732.911   27990.348   \n",
       "\n",
       "    HourSin  WeekCos CBWD  \n",
       "0  0.000000      1.0   NW  \n",
       "1  0.258819      1.0   NW  \n",
       "2  0.500000      1.0   NW  \n",
       "3  0.707107      1.0   NW  \n",
       "4  0.866025      1.0   NW  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = load_network_traffic(return_X_y=False)\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the datset, ```TimeStamp``` is the name of time column, time frequency is hour (H). The target variable (i.e., forecast variable) contains six columns, ```Val_1```, ```Val_2```, ```Val_3```, ```Val_4```, ```Val_5``` and ```Val_6``` respectively. The covariables are ```HourSin```, ```WeekCos```, and ```CBWD```, where ```HourSin``` and ```WeekCos``` are numeric variables and ```CBWD``` is a category variable, which is a multivariable time series task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Classification/Regression Data Format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1 Data Format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Different from the forecasting task, the input data for classification and regression tasks are ```nested DataFrame```, which means the variations over a time segment are located in one cell. See example below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    " var_col_0       var_col_1   ...    var_col_n     target\n",
    "x,x,x,...,x     x,x,x,...,x        x,x,x,...,x      0\n",
    "x,x,x,...,x     x,x,x,...,x        x,x,x,...,x      0\n",
    "x,x,x,...,x     x,x,x,...,x        x,x,x,...,x      1\n",
    "x,x,x,...,x     x,x,x,...,x        x,x,x,...,x      1\n",
    "x,x,x,...,x     x,x,x,...,x        x,x,x,...,x      1\n",
    "x,x,x,...,x     x,x,x,...,x        x,x,x,...,x      2\n",
    "x,x,x,...,x     x,x,x,...,x        x,x,x,...,x      2\n",
    "x,x,x,...,x     x,x,x,...,x        x,x,x,...,x      2\n",
    "x,x,x,...,x     x,x,x,...,x        x,x,x,...,x      2\n",
    "```\n",
    "\n",
    "where，x,x,x,...,x stands for a sample in len(x,x,x... ,x) time segment of length the fluctuation of a variable over time. (x) is the value of a variable at a time. ```target``` represents the label of the sample."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    " \n",
    " - The goal of classification and regression tasks is to identify the behavior of a sample, so it is different from the data form of the forecasting task. Each row of the forecasting data represents the value of each variable at a time point, while each row of the classification or regression data represents a sample. In addition to each cell, namely *x, x, x... , x* represents a sample in len(x, x, x... , x) a time segment of length in which a variable fluctuates over time. Each sample discriminates the category (classification) or value (regression) of target according to the sequence behavior of each feature variable.\n",
    " - Intuitively, ```pandas.DadaFrame``` is a two-dimensional data table, each cell stores a value, and now we store a sequence to nest three-dimensional data in a two-dimensional data table, which is why we call it a ```nested DataFrame```.\n",
    " - The goal of classification or regression tasks is to discriminate the category or behavior of each sample, so the trend of the data is the key characteristic. For simplicity, we omit the information of TimeStamp when storing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2 Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2.2.1 Generate a Random Dataset for Tme Series Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>var_0</th>\n",
       "      <th>var_1</th>\n",
       "      <th>var_2</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0    0.887165\n",
       "1    1.340856\n",
       "2    0.325697\n",
       "3   ...</td>\n",
       "      <td>0   -0.289942\n",
       "1    0.919971\n",
       "2   -1.867462\n",
       "3   ...</td>\n",
       "      <td>0   -1.354841\n",
       "1    0.083280\n",
       "2    0.214305\n",
       "3   ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0    1.950219\n",
       "1    0.016051\n",
       "2   -1.621092\n",
       "3   ...</td>\n",
       "      <td>0   -0.803164\n",
       "1    0.576465\n",
       "2    0.010610\n",
       "3   ...</td>\n",
       "      <td>0   -0.248131\n",
       "1   -0.923721\n",
       "2   -0.691555\n",
       "3   ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0    1.216039\n",
       "1   -0.127643\n",
       "2   -0.337693\n",
       "3   ...</td>\n",
       "      <td>0    0.106087\n",
       "1    1.853251\n",
       "2   -1.428858\n",
       "3   ...</td>\n",
       "      <td>0   -1.169346\n",
       "1    0.207621\n",
       "2    0.137396\n",
       "3   ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0   -1.258965\n",
       "1   -0.151087\n",
       "2   -0.150937\n",
       "3   ...</td>\n",
       "      <td>0    0.132244\n",
       "1   -0.230362\n",
       "2   -0.040706\n",
       "3   ...</td>\n",
       "      <td>0    1.060503\n",
       "1    0.634884\n",
       "2    0.011422\n",
       "3   ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0    0.797132\n",
       "1    1.775923\n",
       "2   -1.491301\n",
       "3   ...</td>\n",
       "      <td>0    0.553663\n",
       "1    1.031838\n",
       "2   -0.555002\n",
       "3   ...</td>\n",
       "      <td>0   -0.660046\n",
       "1    1.278639\n",
       "2   -1.011061\n",
       "3   ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0   -0.253653\n",
       "1   -0.159006\n",
       "2    0.378620\n",
       "3   ...</td>\n",
       "      <td>0   -0.079186\n",
       "1    0.423877\n",
       "2   -0.805016\n",
       "3   ...</td>\n",
       "      <td>0   -1.534051\n",
       "1   -0.101453\n",
       "2    0.279004\n",
       "3   ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               var_0  \\\n",
       "0  0    0.887165\n",
       "1    1.340856\n",
       "2    0.325697\n",
       "3   ...   \n",
       "1  0    1.950219\n",
       "1    0.016051\n",
       "2   -1.621092\n",
       "3   ...   \n",
       "2  0    1.216039\n",
       "1   -0.127643\n",
       "2   -0.337693\n",
       "3   ...   \n",
       "3  0   -1.258965\n",
       "1   -0.151087\n",
       "2   -0.150937\n",
       "3   ...   \n",
       "4  0    0.797132\n",
       "1    1.775923\n",
       "2   -1.491301\n",
       "3   ...   \n",
       "5  0   -0.253653\n",
       "1   -0.159006\n",
       "2    0.378620\n",
       "3   ...   \n",
       "\n",
       "                                               var_1  \\\n",
       "0  0   -0.289942\n",
       "1    0.919971\n",
       "2   -1.867462\n",
       "3   ...   \n",
       "1  0   -0.803164\n",
       "1    0.576465\n",
       "2    0.010610\n",
       "3   ...   \n",
       "2  0    0.106087\n",
       "1    1.853251\n",
       "2   -1.428858\n",
       "3   ...   \n",
       "3  0    0.132244\n",
       "1   -0.230362\n",
       "2   -0.040706\n",
       "3   ...   \n",
       "4  0    0.553663\n",
       "1    1.031838\n",
       "2   -0.555002\n",
       "3   ...   \n",
       "5  0   -0.079186\n",
       "1    0.423877\n",
       "2   -0.805016\n",
       "3   ...   \n",
       "\n",
       "                                               var_2  y  \n",
       "0  0   -1.354841\n",
       "1    0.083280\n",
       "2    0.214305\n",
       "3   ...  0  \n",
       "1  0   -0.248131\n",
       "1   -0.923721\n",
       "2   -0.691555\n",
       "3   ...  0  \n",
       "2  0   -1.169346\n",
       "1    0.207621\n",
       "2    0.137396\n",
       "3   ...  1  \n",
       "3  0    1.060503\n",
       "1    0.634884\n",
       "2    0.011422\n",
       "3   ...  1  \n",
       "4  0   -0.660046\n",
       "1    1.278639\n",
       "2   -1.011061\n",
       "3   ...  2  \n",
       "5  0   -1.534051\n",
       "1   -0.101453\n",
       "2    0.279004\n",
       "3   ...  2  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "size=10\n",
    "\n",
    "df = pd.DataFrame({\n",
    "    'var_0': [pd.Series(np.random.normal(size=size)), pd.Series(np.random.normal(size=size)),\n",
    "              pd.Series(np.random.normal(size=size)), pd.Series(np.random.normal(size=size)),\n",
    "              pd.Series(np.random.normal(size=size)), pd.Series(np.random.normal(size=size))],\n",
    "    'var_1': [pd.Series(np.random.normal(size=size)), pd.Series(np.random.normal(size=size)),\n",
    "              pd.Series(np.random.normal(size=size)), pd.Series(np.random.normal(size=size)),\n",
    "              pd.Series(np.random.normal(size=size)), pd.Series(np.random.normal(size=size))],\n",
    "    'var_2': [pd.Series(np.random.normal(size=size)), pd.Series(np.random.normal(size=size)),\n",
    "              pd.Series(np.random.normal(size=size)), pd.Series(np.random.normal(size=size)),\n",
    "              pd.Series(np.random.normal(size=size)), pd.Series(np.random.normal(size=size))],\n",
    "    'y': [0, 0, 1, 1, 2, 2], \n",
    "})\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output shows that:\n",
    "\n",
    "- The name of the target variable is 'y'; \n",
    "- The names of the feature variables are 'var_0',  'var_1',  'var_2';\n",
    "- It's a multivariate classification task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2.2.2 Import build-in Classification Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hyperts.datasets import load_arrow_head, load_basic_motions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Var_1</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0     -1.9630\n",
       "1     -1.9578\n",
       "2     -1.9561\n",
       "3   ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0     -1.7746\n",
       "1     -1.7740\n",
       "2     -1.7766\n",
       "3   ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0     -1.8660\n",
       "1     -1.8420\n",
       "2     -1.8350\n",
       "3   ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0     -2.0738\n",
       "1     -2.0733\n",
       "2     -2.0446\n",
       "3   ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0     -1.7463\n",
       "1     -1.7413\n",
       "2     -1.7227\n",
       "3   ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               Var_1 target\n",
       "0  0     -1.9630\n",
       "1     -1.9578\n",
       "2     -1.9561\n",
       "3   ...      0\n",
       "1  0     -1.7746\n",
       "1     -1.7740\n",
       "2     -1.7766\n",
       "3   ...      1\n",
       "2  0     -1.8660\n",
       "1     -1.8420\n",
       "2     -1.8350\n",
       "3   ...      2\n",
       "3  0     -2.0738\n",
       "1     -2.0733\n",
       "2     -2.0446\n",
       "3   ...      0\n",
       "4  0     -1.7463\n",
       "1     -1.7413\n",
       "2     -1.7227\n",
       "3   ...      1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df0 = load_arrow_head(return_X_y=False)\n",
    "df0.head() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['0', '1', '2'], dtype=object)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df0.target.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this dataset, ```target``` is the target column, containing three classes, ['0', '1', '2']. ```Var_1``` is a feature variable, there is only one, so it is a univariate multi-classification task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Var_1</th>\n",
       "      <th>Var_2</th>\n",
       "      <th>Var_3</th>\n",
       "      <th>Var_4</th>\n",
       "      <th>Var_5</th>\n",
       "      <th>Var_6</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0     0.079106\n",
       "1     0.079106\n",
       "2    -0.903497\n",
       "3...</td>\n",
       "      <td>0     0.394032\n",
       "1     0.394032\n",
       "2    -3.666397\n",
       "3...</td>\n",
       "      <td>0     0.551444\n",
       "1     0.551444\n",
       "2    -0.282844\n",
       "3...</td>\n",
       "      <td>0     0.351565\n",
       "1     0.351565\n",
       "2    -0.095881\n",
       "3...</td>\n",
       "      <td>0     0.023970\n",
       "1     0.023970\n",
       "2    -0.319605\n",
       "3...</td>\n",
       "      <td>0     0.633883\n",
       "1     0.633883\n",
       "2     0.972131\n",
       "3...</td>\n",
       "      <td>standing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0     0.377751\n",
       "1     0.377751\n",
       "2     2.952965\n",
       "3...</td>\n",
       "      <td>0    -0.610850\n",
       "1    -0.610850\n",
       "2     0.970717\n",
       "3...</td>\n",
       "      <td>0    -0.147376\n",
       "1    -0.147376\n",
       "2    -5.962515\n",
       "3...</td>\n",
       "      <td>0    -0.103872\n",
       "1    -0.103872\n",
       "2    -7.593275\n",
       "3...</td>\n",
       "      <td>0    -0.109198\n",
       "1    -0.109198\n",
       "2    -0.697804\n",
       "3...</td>\n",
       "      <td>0    -0.037287\n",
       "1    -0.037287\n",
       "2    -2.865789\n",
       "3...</td>\n",
       "      <td>standing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0    -0.813905\n",
       "1    -0.813905\n",
       "2    -0.424628\n",
       "3...</td>\n",
       "      <td>0     0.825666\n",
       "1     0.825666\n",
       "2    -1.305033\n",
       "3...</td>\n",
       "      <td>0     0.032712\n",
       "1     0.032712\n",
       "2     0.826170\n",
       "3...</td>\n",
       "      <td>0     0.021307\n",
       "1     0.021307\n",
       "2    -0.372872\n",
       "3...</td>\n",
       "      <td>0     0.122515\n",
       "1     0.122515\n",
       "2    -0.045277\n",
       "3...</td>\n",
       "      <td>0     0.775041\n",
       "1     0.775041\n",
       "2     0.383526\n",
       "3...</td>\n",
       "      <td>standing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0     0.289855\n",
       "1     0.289855\n",
       "2    -0.669185\n",
       "3...</td>\n",
       "      <td>0     0.284130\n",
       "1     0.284130\n",
       "2    -0.210466\n",
       "3...</td>\n",
       "      <td>0     0.213680\n",
       "1     0.213680\n",
       "2     0.252267\n",
       "3...</td>\n",
       "      <td>0    -0.314278\n",
       "1    -0.314278\n",
       "2     0.018644\n",
       "3...</td>\n",
       "      <td>0     0.074574\n",
       "1     0.074574\n",
       "2     0.007990\n",
       "3...</td>\n",
       "      <td>0    -0.079901\n",
       "1    -0.079901\n",
       "2     0.237040\n",
       "3...</td>\n",
       "      <td>standing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0    -0.123238\n",
       "1    -0.123238\n",
       "2    -0.249547\n",
       "3...</td>\n",
       "      <td>0     0.379341\n",
       "1     0.379341\n",
       "2     0.541501\n",
       "3...</td>\n",
       "      <td>0    -0.286006\n",
       "1    -0.286006\n",
       "2     0.208420\n",
       "3...</td>\n",
       "      <td>0    -0.098545\n",
       "1    -0.098545\n",
       "2    -0.023970\n",
       "3...</td>\n",
       "      <td>0     0.058594\n",
       "1     0.058594\n",
       "2     0.175783\n",
       "3...</td>\n",
       "      <td>0    -0.074574\n",
       "1    -0.074574\n",
       "2     0.114525\n",
       "3...</td>\n",
       "      <td>standing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               Var_1  \\\n",
       "0  0     0.079106\n",
       "1     0.079106\n",
       "2    -0.903497\n",
       "3...   \n",
       "1  0     0.377751\n",
       "1     0.377751\n",
       "2     2.952965\n",
       "3...   \n",
       "2  0    -0.813905\n",
       "1    -0.813905\n",
       "2    -0.424628\n",
       "3...   \n",
       "3  0     0.289855\n",
       "1     0.289855\n",
       "2    -0.669185\n",
       "3...   \n",
       "4  0    -0.123238\n",
       "1    -0.123238\n",
       "2    -0.249547\n",
       "3...   \n",
       "\n",
       "                                               Var_2  \\\n",
       "0  0     0.394032\n",
       "1     0.394032\n",
       "2    -3.666397\n",
       "3...   \n",
       "1  0    -0.610850\n",
       "1    -0.610850\n",
       "2     0.970717\n",
       "3...   \n",
       "2  0     0.825666\n",
       "1     0.825666\n",
       "2    -1.305033\n",
       "3...   \n",
       "3  0     0.284130\n",
       "1     0.284130\n",
       "2    -0.210466\n",
       "3...   \n",
       "4  0     0.379341\n",
       "1     0.379341\n",
       "2     0.541501\n",
       "3...   \n",
       "\n",
       "                                               Var_3  \\\n",
       "0  0     0.551444\n",
       "1     0.551444\n",
       "2    -0.282844\n",
       "3...   \n",
       "1  0    -0.147376\n",
       "1    -0.147376\n",
       "2    -5.962515\n",
       "3...   \n",
       "2  0     0.032712\n",
       "1     0.032712\n",
       "2     0.826170\n",
       "3...   \n",
       "3  0     0.213680\n",
       "1     0.213680\n",
       "2     0.252267\n",
       "3...   \n",
       "4  0    -0.286006\n",
       "1    -0.286006\n",
       "2     0.208420\n",
       "3...   \n",
       "\n",
       "                                               Var_4  \\\n",
       "0  0     0.351565\n",
       "1     0.351565\n",
       "2    -0.095881\n",
       "3...   \n",
       "1  0    -0.103872\n",
       "1    -0.103872\n",
       "2    -7.593275\n",
       "3...   \n",
       "2  0     0.021307\n",
       "1     0.021307\n",
       "2    -0.372872\n",
       "3...   \n",
       "3  0    -0.314278\n",
       "1    -0.314278\n",
       "2     0.018644\n",
       "3...   \n",
       "4  0    -0.098545\n",
       "1    -0.098545\n",
       "2    -0.023970\n",
       "3...   \n",
       "\n",
       "                                               Var_5  \\\n",
       "0  0     0.023970\n",
       "1     0.023970\n",
       "2    -0.319605\n",
       "3...   \n",
       "1  0    -0.109198\n",
       "1    -0.109198\n",
       "2    -0.697804\n",
       "3...   \n",
       "2  0     0.122515\n",
       "1     0.122515\n",
       "2    -0.045277\n",
       "3...   \n",
       "3  0     0.074574\n",
       "1     0.074574\n",
       "2     0.007990\n",
       "3...   \n",
       "4  0     0.058594\n",
       "1     0.058594\n",
       "2     0.175783\n",
       "3...   \n",
       "\n",
       "                                               Var_6    target  \n",
       "0  0     0.633883\n",
       "1     0.633883\n",
       "2     0.972131\n",
       "3...  standing  \n",
       "1  0    -0.037287\n",
       "1    -0.037287\n",
       "2    -2.865789\n",
       "3...  standing  \n",
       "2  0     0.775041\n",
       "1     0.775041\n",
       "2     0.383526\n",
       "3...  standing  \n",
       "3  0    -0.079901\n",
       "1    -0.079901\n",
       "2     0.237040\n",
       "3...  standing  \n",
       "4  0    -0.074574\n",
       "1    -0.074574\n",
       "2     0.114525\n",
       "3...  standing  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = load_basic_motions(return_X_y=False)\n",
    "df1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['standing', 'running', 'walking', 'badminton'], dtype=object)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.target.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this dataset, ```target``` is the target column, containing four classes ['standing', 'running', 'walking', 'badminton'], ```Var_1```, ```Var_2```, ```Var_3```, ```Var_4```, ```Var_5```, ```Var_6``` are feature variables, a total of six. Therefore, it is a multivariate multiclassification task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.3 3-D Numpy.array to 2-D pandas.DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, the acquired data is in the form of ``numpy.array``. We need to convert them to the nested ``pandas.DataFrame``. For example, we have the numpy data as below: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "nb_samples = 100\n",
    "series_length = 72\n",
    "nb_variables = 6\n",
    "nb_classes = 4\n",
    "\n",
    "X = np.random.normal(size=nb_samples*series_length*nb_variables).reshape(nb_samples, series_length, nb_variables)\n",
    "y = np.random.randint(low=0, high=nb_classes, size=nb_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((100, 72, 6), (100,), array([0, 1, 2, 3]))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape, y.shape, np.unique(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset contains 100 samples. Each sample has 6 feature variables. And each variable has measurement data taken at 72 time indices. The target variable *y* has 4 categories."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HyperTS porvides a function ``from_3d_array_to_nested_df``, that could automaticlly convert 3d array to required nested dataframe. See example below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from hyperts.toolbox import from_3d_array_to_nested_df\n",
    "\n",
    "df_X = from_3d_array_to_nested_df(data=X)\n",
    "df_y = pd.DataFrame({'y': y})\n",
    "df = pd.concat([df_X, df_y], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Var_0</th>\n",
       "      <th>Var_1</th>\n",
       "      <th>Var_2</th>\n",
       "      <th>Var_3</th>\n",
       "      <th>Var_4</th>\n",
       "      <th>Var_5</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0    -0.038029\n",
       "1     2.407684\n",
       "2     1.000662\n",
       "3...</td>\n",
       "      <td>0     0.947376\n",
       "1    -0.666230\n",
       "2    -0.673983\n",
       "3...</td>\n",
       "      <td>0     0.294960\n",
       "1    -0.188034\n",
       "2     0.791917\n",
       "3...</td>\n",
       "      <td>0    -0.260460\n",
       "1     1.190756\n",
       "2     0.183264\n",
       "3...</td>\n",
       "      <td>0     1.456938\n",
       "1     0.773571\n",
       "2     1.834219\n",
       "3...</td>\n",
       "      <td>0     0.028061\n",
       "1    -1.742055\n",
       "2    -0.520803\n",
       "3...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0     0.943434\n",
       "1     0.710932\n",
       "2    -0.717793\n",
       "3...</td>\n",
       "      <td>0    -0.816563\n",
       "1    -0.206350\n",
       "2    -0.022789\n",
       "3...</td>\n",
       "      <td>0     1.604838\n",
       "1    -1.233925\n",
       "2     0.068006\n",
       "3...</td>\n",
       "      <td>0     1.471666\n",
       "1     0.215581\n",
       "2    -1.120550\n",
       "3...</td>\n",
       "      <td>0     0.426381\n",
       "1     0.911374\n",
       "2     1.202849\n",
       "3...</td>\n",
       "      <td>0    -0.362679\n",
       "1    -0.585743\n",
       "2    -0.212023\n",
       "3...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0    -0.957144\n",
       "1     0.126544\n",
       "2    -0.962839\n",
       "3...</td>\n",
       "      <td>0     0.085112\n",
       "1     0.704056\n",
       "2     1.869679\n",
       "3...</td>\n",
       "      <td>0    -2.551469\n",
       "1    -2.025863\n",
       "2    -1.088073\n",
       "3...</td>\n",
       "      <td>0    -0.790131\n",
       "1     0.756131\n",
       "2     0.676036\n",
       "3...</td>\n",
       "      <td>0    -1.184995\n",
       "1    -0.160961\n",
       "2    -0.864118\n",
       "3...</td>\n",
       "      <td>0     0.630483\n",
       "1     1.281456\n",
       "2    -1.885355\n",
       "3...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0     0.745456\n",
       "1    -0.608358\n",
       "2    -2.248564\n",
       "3...</td>\n",
       "      <td>0    -0.708104\n",
       "1    -0.323268\n",
       "2    -1.456815\n",
       "3...</td>\n",
       "      <td>0     0.259860\n",
       "1     0.600146\n",
       "2     1.458288\n",
       "3...</td>\n",
       "      <td>0     0.628735\n",
       "1     0.064484\n",
       "2     1.336053\n",
       "3...</td>\n",
       "      <td>0     0.568531\n",
       "1    -0.059870\n",
       "2    -0.230826\n",
       "3...</td>\n",
       "      <td>0    -0.845890\n",
       "1     0.848568\n",
       "2    -0.662570\n",
       "3...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0     1.377579\n",
       "1    -0.915978\n",
       "2    -1.185956\n",
       "3...</td>\n",
       "      <td>0     1.276868\n",
       "1    -1.068383\n",
       "2    -0.765125\n",
       "3...</td>\n",
       "      <td>0    -0.252029\n",
       "1    -1.035151\n",
       "2    -0.677545\n",
       "3...</td>\n",
       "      <td>0    -0.033123\n",
       "1     0.387280\n",
       "2    -0.658591\n",
       "3...</td>\n",
       "      <td>0    -0.007119\n",
       "1    -0.546834\n",
       "2    -1.624636\n",
       "3...</td>\n",
       "      <td>0     0.060272\n",
       "1     1.340591\n",
       "2     0.678717\n",
       "3...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               Var_0  \\\n",
       "0  0    -0.038029\n",
       "1     2.407684\n",
       "2     1.000662\n",
       "3...   \n",
       "1  0     0.943434\n",
       "1     0.710932\n",
       "2    -0.717793\n",
       "3...   \n",
       "2  0    -0.957144\n",
       "1     0.126544\n",
       "2    -0.962839\n",
       "3...   \n",
       "3  0     0.745456\n",
       "1    -0.608358\n",
       "2    -2.248564\n",
       "3...   \n",
       "4  0     1.377579\n",
       "1    -0.915978\n",
       "2    -1.185956\n",
       "3...   \n",
       "\n",
       "                                               Var_1  \\\n",
       "0  0     0.947376\n",
       "1    -0.666230\n",
       "2    -0.673983\n",
       "3...   \n",
       "1  0    -0.816563\n",
       "1    -0.206350\n",
       "2    -0.022789\n",
       "3...   \n",
       "2  0     0.085112\n",
       "1     0.704056\n",
       "2     1.869679\n",
       "3...   \n",
       "3  0    -0.708104\n",
       "1    -0.323268\n",
       "2    -1.456815\n",
       "3...   \n",
       "4  0     1.276868\n",
       "1    -1.068383\n",
       "2    -0.765125\n",
       "3...   \n",
       "\n",
       "                                               Var_2  \\\n",
       "0  0     0.294960\n",
       "1    -0.188034\n",
       "2     0.791917\n",
       "3...   \n",
       "1  0     1.604838\n",
       "1    -1.233925\n",
       "2     0.068006\n",
       "3...   \n",
       "2  0    -2.551469\n",
       "1    -2.025863\n",
       "2    -1.088073\n",
       "3...   \n",
       "3  0     0.259860\n",
       "1     0.600146\n",
       "2     1.458288\n",
       "3...   \n",
       "4  0    -0.252029\n",
       "1    -1.035151\n",
       "2    -0.677545\n",
       "3...   \n",
       "\n",
       "                                               Var_3  \\\n",
       "0  0    -0.260460\n",
       "1     1.190756\n",
       "2     0.183264\n",
       "3...   \n",
       "1  0     1.471666\n",
       "1     0.215581\n",
       "2    -1.120550\n",
       "3...   \n",
       "2  0    -0.790131\n",
       "1     0.756131\n",
       "2     0.676036\n",
       "3...   \n",
       "3  0     0.628735\n",
       "1     0.064484\n",
       "2     1.336053\n",
       "3...   \n",
       "4  0    -0.033123\n",
       "1     0.387280\n",
       "2    -0.658591\n",
       "3...   \n",
       "\n",
       "                                               Var_4  \\\n",
       "0  0     1.456938\n",
       "1     0.773571\n",
       "2     1.834219\n",
       "3...   \n",
       "1  0     0.426381\n",
       "1     0.911374\n",
       "2     1.202849\n",
       "3...   \n",
       "2  0    -1.184995\n",
       "1    -0.160961\n",
       "2    -0.864118\n",
       "3...   \n",
       "3  0     0.568531\n",
       "1    -0.059870\n",
       "2    -0.230826\n",
       "3...   \n",
       "4  0    -0.007119\n",
       "1    -0.546834\n",
       "2    -1.624636\n",
       "3...   \n",
       "\n",
       "                                               Var_5  y  \n",
       "0  0     0.028061\n",
       "1    -1.742055\n",
       "2    -0.520803\n",
       "3...  2  \n",
       "1  0    -0.362679\n",
       "1    -0.585743\n",
       "2    -0.212023\n",
       "3...  1  \n",
       "2  0     0.630483\n",
       "1     1.281456\n",
       "2    -1.885355\n",
       "3...  2  \n",
       "3  0    -0.845890\n",
       "1     0.848568\n",
       "2    -0.662570\n",
       "3...  2  \n",
       "4  0     0.060272\n",
       "1     1.340591\n",
       "2     0.678717\n",
       "3...  1  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Anomaly Detection Data Format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to forecasting, the required input data format is a two-dimensional structure (``pandas.DataFrame``), which should contain a TimeStamp column (``time_col``) ,one or more variable columns (``var_col_0``, ``var_col_1``, ``var_col_2``,... ``var_col_n``), and sometimes, one or more covariates columns (``covar_col_0``, ``covar_col_1``, ``covar_col_2``,... ``covar_col_m``). See example below: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "     time_col          var_col_0   var_col_1 ... var_col_n     covar_col_0    covar_col_1 ... covar_col_m\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x \n",
    "xxxx-xx-xx xx:xx:xx        x          x              -              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              -               x\n",
    "xxxx-xx-xx xx:xx:xx        -          x              x              x              x               -\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        x          -              x              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        x          -              x              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              -               x\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        -          -              -              x              x               x\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x\n",
    "        -                  -          -              -              -              -               -\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, the above data can also contain ``ground truth``, which will help in model selection and hyperparameter search. The format is as follow:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "     time_col          var_col_0   var_col_1 ... var_col_n     covar_col_0    covar_col_1 ... covar_col_m       anomaly       \n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x               1 \n",
    "xxxx-xx-xx xx:xx:xx        x          x              -              x              x               x               0\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              -               x               0\n",
    "xxxx-xx-xx xx:xx:xx        -          x              x              x              x               -               0\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x               1\n",
    "xxxx-xx-xx xx:xx:xx        x          -              x              x              x               x               0\n",
    "xxxx-xx-xx xx:xx:xx        x          -              x              x              x               x               0\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              -               x               1\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x               0\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x               0\n",
    "xxxx-xx-xx xx:xx:xx        -          -              -              x              x               x               1\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x               0\n",
    "        -                  -          -              -              -              -               -\n",
    "xxxx-xx-xx xx:xx:xx        x          x              x              x              x               x               0\n",
    "```\n",
    "\n",
    "\n",
    "where ``anomaly`` is anomaly label column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "   When the data has ground truth label, the optimization evalution uses the ground truth. Otherwise, the generated pseudo-label is applied."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
